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IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

In re Application of 
David Mark FROHLICH et al. 
Serial No. Not yet assigned 
Filed: herewith 

For: DIGITAL CAMERA WITH SOUND RECORDING 

PRELIMINARY AMENDMENT 



Assistant Commissioner For Patents 
Washington, D.C. 20231 



Dear Sir: 

Preliminary to examination of the above-referenced application, please amend the 
application: 



IN THE CLAIMS: 

Please amend claims 4, 6, 7, 8, 10, 11, 12, 13, 15, 16, 18, 21, 22 and 23 as follows: 

4. A digital camera as claimed in [any of claims] claim 1 [to 
3], wherein the user interface is adapted such that a user 
can select a first and second view of the image files, 
wherein in the first view a plurality of sound passage 
files, still image files or linked groups are displayed in 
an index format for review, and wherein in the second view 
an individual sound passage file, still image file or linked 
group is displayed for editing. 

6- A digital camera as claimed in claim 4 [or claim 5], wherein 
in the second view a further sound passage or still image may 
be captured and linked automatically to the individual sound 
passage file, still image file, or linked group displayed. 
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7. A digital camera as claimed in [any preceding] claim 1, 
wherein the digital camera is adapted such that a still 
image file may be associated with a particular point of time 
or period of time in the sound passage of a sound passage 
file. 

8. A digital camera as claimed in [any preceding] claim 1, 
wherein the digital camera is adapted such that the camera 
apparatus may capture one or more still images at the same 
time that the sound recording apparatus is capturing a sound 
passage . 

10. A digital camera as claimed in [any preceding] claim 1, 
further comprising a time recording means to mark a captured 
still image or a captured sound passage with the time of 
recording . 

11. A digital camera as claimed in claim 10 [where dependent on 
claim 8], wherein the digital camera is adapted such that the 
camera apparatus may capture one or more still images at the 
same time that the sound recording apparatus is capturing a 
sound passage, and the time recording means is adapted to 
mark the time of capture of a captured still image relative 
to a contemporaneously recorded sound passage . 

12. A digital camera as claimed in [any preceding] claim 1, 
wherein the user interface is adapted for selective deletion 
by a user of a still image file, a sound passage file or 
part of a sound passage file, or a linked group. 

13. A digital camera as claimed in [any preceding] claim 1, 
wherein the user interface is adapted such that sound 
passage files and still image files of a linked group are 
viewable, selectively, either as part of the linked group or 
independently of the linked group. 
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15. A digital camera as claimed in [any preceding] claim • 1, 
wherein the memory retains linked groups as separate sound 
passage fileS;. still image files, and one or more index 
files containing linking information. 

16. A digital camera as claimed in [any preceding] claim 1^ 
wherein the memory retains linked groups as multimedia 
files, containing sound passage information, still image 
information, and linking information. 

18. A digital camera as claimed in [any preceding] claim 1, 
further comprising a further data recording device for 
recording of another data type, and wherein the memory 
device is adapted to record the other data type as a further 
data type file, and wherein the user interface is adapted 
for selectively linking one or more further data type files 
into a linked group, and for selectively unlinking one or 
more further data type files from a linked group. 

21. A digital camera as claimed in [any preceding] claim _1, 
wherein the sound recording apparatus and the camera 
apparatus are arranged for data capture in a substantially 
common direction . 

22. A digital camera as claimed in [any preceding] claim 1^, 
wherein the memory is a flash memory device. 

23. A digital camera as claimed in [any preceding] claim 1, 
wherein the digital camera is adapted to be handheld by a 
user for still image and sound passage capture. 
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REMARKS 



The above-referenced application is amended to delete the multiple dependencies of claims 
4, 6, 7, 8, 10, 11, 12, 13, 15, 16, 18, 21, 22 and 23 to avoid the multiple dependent claim filing fee. 



Respectfully submitted, 

LOWE HAUPTMAN OILMAN & BERNER, LLP 



Allan M. Low* 
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The present invention relates to a digital camera adapted to allow recording of sound 
5 as well as still images. In particular, the invention relates to a digital camera 
arrangement designed to support the construction of different combinations of sound 
and images. 

It is known that the combination of a still image and sound is a valuable one for the 
10 user, not only where the sound is provided as atmotation, but also where the sound is 

ambient at the time of recording tiie still image. Examples of systems adapted for 

recording both still images and sound together are disclosed in US Patents Nos. 

3439598, 4270853, 4270854, 4905029 and 5276472. A system for display and 

playback of composite sound and still image is set out in European Patent AppUcation 
15 Publication No. 0827018. This combination of sound and still image is also 

obtainable from certain digital video cameras which have capacity to capture still 

images - an example is the Hitachi MP-EGl A. 

A camera operating system which supports augmentation of still images with sound is 
20 Digita, provided by Flashpoint Technology, Inc. Digita provides a menu-driven 
interface, and also scripts which allow construction of composite items at the camera 
(such as a page written in HTML with camera-captured image inserts). 

In existing products, the sound capture is generally optimised for annotation, rather 
25 than for recording of ambient soimd. Accordingly, the microphone is directed 
towards the user rather than to the imaged scene, the sound is recorded after the image 
is captured, and the combination of media is limited to one sound clip per image. 
Although capture of ambient sound is a known use of cameras with sound capture 
features, existing arrangements still provide a simple relationship of association 
30 between a single still image and a single piece of soimd. 

It is desirable to improve the capture of soimd together with still images to allow a 
rich combination of the two which will provide particular benefits to the user. 
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Summary of Invention 

Accordingly, the invention provides a digital camera comprising: camera apparatus 
for capturing still images; sound recording apparatus for capturing sound passages; a 
5 memory for retaining still images as still image files and sound passages as sound 
passage files; and a user interface adapted for selectively linking one or more still 
image files with one or more sound passage files to form a linked group, and for 
selectively unlinking one or more files from a linked group. 

10 Still image files and soimd passage files may be files of different types, in which case 
selective linking between files may be recorded in an index file containing all linkage 
information. Alternatively, linked groups may have a specific multunedia filetype 
supporting still image files, sound passage files, and links therebetween: in this case, 
still images and sound passages may also themselves be provided in this filetype, as it 

15 possesses the inherent capacity to be either a still image file, a sound passage file, or 
both. 

By treating the images and sound as equivalent forms of data in this way, rather than 
merely considering a sound passage as an optional addition to a still image, richer 

20 possibilities for creation and editing of associated sound and image data become 
available. This equivalence makes selection of sound and image to be matched 
together straightforward. Advantageously, it also allows for creation of more 
complex objects, with a Unked group containing any of one still image file with one or 
more soxmd passage files; one sound passage file with one or more still image files; or 

25 even a plurality of both sound passage files and still image files. This flexibility is 
particxilarly important where ambient soxmd is to be captured: for example, a desirable 
approach may be to record the sound, and during recording of the sound to capture a 
large number of still images, deciding at a later point which images to keep. With a 
system in which sound is provided through annotation of a single image, this desirable 

30 mode of operating is simply not possible. 

Advantageously, the user interface is adapted such that a user can select a first and 
second view of the image files, wherein in the first view a plurality of sound passage 
files, still image files or linked groups are displayed in an index format for browsing. 
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and wherein in the second view a single sound passage file, still image file or linked 
group is displayed for review and editing (including modification of any link). It is 
helpfixl if in this second view an additional sound passage file, still image file or 
linked group is also displayable, and the user interface is adapted to allow linkage 

5 between the displayed files to form a new Unked group. Another desirable feature is 
for the sound passage files and still image files of a linked group to be viewable, 
selectively, either as part of the linked group or independently of the linked group. 
Another usefiil feature is for the user interface is adapted so that the sound passage 
files, the still image files, and the linked objects are all presentable to the user as a 

10 separate sequence, and that the user may select between these separate sequences for 
browsiag. This collection of features allows effective browsing through a collection 
of captured data, and effective combination and review of the data collected. 

It is particularly usefiil, especially in the case of more complex linked groups, for the 
15 digital camera to be adapted such that a still image file may be associated with a 
particular point of time or period of time in the sound passage of a sound passage file. 
This is valuable if images and sound passages can be captured contemporaneousl)^ In 
connection with this, it is desirable for there to be a means to mark a captured still 
image or a captured sound passage with the time of recording. This time stamping 
20 can either be absolute, or in the case of contemporaneous recording, relative (that is, 
an image is stamped with its time of recording relative to the start of the sound 
passage). Contemporaneous recordings of sound and still images may be provided 
automatically as linked groups. This structure allows valuable forms of image 
association to be created, such as an assemblage of still images appearing in a 
25 determined succession and spacing against a common passage of ambient sound. 

In a development of this approach, embodiments of a camera may include a fiirther 
data recording device for recording of another data type, and wherein the memory 
device is adapted to record the other data type as a further data type file, and wherein 
30 the user interface is adapted for selectively linking one or more fiirther data type files 
into a linked group, and for selectively xmlinldng one or more fiirther data type files 
from a linked group. These fiirther data types may be graphic annotation (for 
example, handwriting input) or video cHps. 



Description of Figures and Specific Embodiment 

Specific embodiments of the invention are described below, by way of example, with 
reference to the accompanying drawings, in which: 

5 

Figure 1 shows a system for the capture and reproduction of sound and still images 
according to an arrangement proposed by the present appUcant; 

Figures 2a, 2b and 2c show fi*ont (image and sound capture) side, rear (user interface 
10 side) and top (capture button) views of a digital camera in accordance with m 
embodiment of the invention; 

Figures 3a to 3c show the user interface display of Figure 2 in editing mode, and 
illustrate selection of different data elements; 

15 

Figure 4 shows the user interface display of Figure 2 in editing mode, and shows an 
audiophoto consisting of a sound passage with three attached images; 

Figures 5a to 5c show the user interface display of Figure 2 at three different stages 
20 during the playing of the audiophoto of Figure 4; 

Figmre 6 shows the user interface display of Figure 2 in editing mode showing an 
index image of captured data; and 

25 Figures 7a and 7b show the user interface display of Figure 2 in capture mode, and 
illustrate respectively a viewfinder display and an index display. 

Basic components in a system for providing an image and related sound are shown in 
30 Figure 1. A camera 1 comprises a conventional digital camera mechanism for 
capturing a still image together with a digital sound recording device for recording a 
passage of sound. Sound recording means are conventional: a microphone 101, an 
automatic gain controller 102, an analog/digital converter 103 and a codec 106 - 
where playback is also available, as in the embodiment to be described below, a 
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loudspeaker 105 and a digital/analog converter 104 are also be provided. A memory 
107 for storage of audio and image information is also required. 

It should be noted that althou^ the term "image" is used here, it does not refer merely 
5 to a pictorial image. It may also relate to, for example, text captured in an image 
format. Likewise, when the term "camera" is used, other devices for capturing a still 
image can be understood to fall within its scope in the context of the present 
invention. 

10 Whereas the process of image and sound capture may be entirely analog, in which 
case both an analog image recording medium (such as film) and an analog sound 
recording medium are provided at the camera, in the context of the present invention 
digital capture of both sound and image is used. Both image and sound are digitized 
on capture and stored in an appropriate storage medium (for example on a 

15 conventional flash memory or an Iomega "Clik" disk ("Clik" is a trade mark of 
Iomega Corporation)). In this event, the digital data can be provided to a digital 
processing means (such as personal computer 2). This is advantageous, as it allows 
for easy editing of both image and sound data. The image data can then be provided 
fi-om the digital processing means as a printed image, and the passage of sound 

20 attached thereto in a digital electronic storage medium, such as a flash memory. This 
is best achieved by means of a printer 3. In printer 3, the image is printed in 
conventional manner and a sound reproduction device with the passage of sound 
recorded thereon is attached to the image. Advantageously, recording and attachment 
of sound is done in the printer itself, but this step may be carried out separately (for 

25 example, at the sound recording device 4, as is described further below). 
Altematively, it is possible in some embodiments for the image data and optionally 
the soimd data to be provided directly from the camera to the printer (for example by 
infra red data transfer using technologies currently known for infra red printing from 
personal computers - such as xmder the IrTranP standard): these arrangements may 

30 require the processor in camera 1 to be provided with a printer driver for the 
appropriate printer. 

The result is a tangible representation of the image with an electronic storage medium 
having the passage of sound recorded thereon attached to it: this is here termed an 
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"audiophoto". To play the passage of sound, sound reproduction device 4 is 
employed. This can be connected to the electronic storage medium to enable 
information stored in the electronic storage device to be transferred to the sound 
reproduction device, the sound reproduction device 4 containing means to convert the 
5 information received from the electronic storage device into the passage of sound. 
However, sound reproduction device 4 is adapted to be detachable or otherwise 
remote from the printed image when no connection between the electronic storage 
device and the sound reproduction device is required. 

10 Further details of these system elements are provided in copending European Patent 
Application No. 98305436.2, the contents of which are incorporated by reference 
herein to the extent permissible by law. A digital camera 1 which is an embodiment 
of the present invention will now be discussed below, with reference to Figures 2a to 
2c. 

15 

Figure 2a shows the front (image capture) side of a digital camera 1 according to an 
embodiment of the present invention. In basic form, digital camera 1 is a 
conventional handheld camera. This camera is optimised for capture of ambient 
sound, rather than annotation of images, so in addition to a conventional digital 

20 camera apparatus for capturing still images shown here by lens 21, the front side of 
the camera 1 also has the microphone 101 of a sound recording apparatus (of the kind 
discussed with reference to Figure 1) for capturing sound passages. In a digital 
camera optimised for annotation, this microphone 101 would be placed on the rear 
(user interface) side of the digital camera 1 - altematively, dual microphones, or 

25 means to reorient the microphone for capture from different directions at different 
times, may be provided. The captured images and sound are eventually stored (in 
formats and associations discussed below in the context of the user interface of the 
digital camera) on a memory for retaining still images as still image files and soimd 
passages as sound passage files. This memory comprises a removable flash memory 

30 card 107 held in a slot on the side of the camera, and released from the slot by a 
button release 22. 

Figure 2b shows the rear (user interface) side of the digital camera 1. The display 23 
is (preferably) an LCD or other flat screen display. The display can be used either for 



image and sound capture (capture mode) or for image and soxmd editing (editing 
mode) - both capture mode and editing mode are discussed further below. The 
display 23 is shown in Figure 2b in editing mode, with an audiophoto image shown 
with three display elements: image element 231, sound element 232, and selection 
5 element 233, all of which will be addressed below in the discussion of editing mode. 
Switch 26 is used to allow the user to toggle between capture mode and editing mode. 
Also provided on the user interface side of the digital camera is loudspeaker 102, so 
that audio playback is directed towards the user. 

10 The user interface controls 24 are provided on the user interface side of the digital 
camera. The user interface controls 24 are used to browse, organise and edit the 
captured images and sound, and in particular to allow selective linking of one or more 
still image files with one or more sound passage files to form a linked group, and for 
selectively unlinking one or more files firom a linked group. The specific fimctions of 
15 different buttons will be described in greater detail in the discussion of the editing 
mode, but a brief summary is as follows: 

data type selection button 241 allows selection of image, sound or both as 
"active"; 

deletion button 242 allows deletion of the current active element; 
20 linking/unlinking button 243 allows selected elements to be linked or 

luilinked; 

marking button 244 allows a soimd clip to be marked (for reasons discussed 
fluther below); 

forward button 245 allows a user to step forward throughout data while 
25 browsing, or to play a sound passage; 

backward button 246 allows a user to step backward throughout data while 
browsing; and 

stop button 247 allows a user to stop playing of a soimd passage. 



30 Figure 2c shows a top view of the camera 1. The top of the camera is where data 
capture controls 25 are located. Data capture controls 25 are essentially conventional: 
there is an image capture button 251 easily operable with the right hand to record an 
image, and a dual state sound capture start button 252 easily operable with the left 



hand to begin recording of sound on a first button press to end recording of sound on a 
second button press. 

Editing mode will now be discussed with reference to Figures 3 to 6. 

5 

Figure 3 a shows an audiophoto in editing mode. The audiophoto consists of a still 
image 231 and a soimd passage (provided as a graphical representation 232 of the 
sound). The sound passage is marked with a tag 31 to mdicate the association of an 
image with it - the position of the tag 31 is indicative of the time along the sound 
10 stream represented by the graphical representation 232. The sound passage is also 
marked with its duration 32. Selection element 233 indicates in Figure 3a that the 
whole audiophoto is selected. The selection element can be toggled between the 
states of audiophoto/sound element/image element by pressing the data type selection 
button 241. 

15 

When the audiophoto is selected as in Figure 3a, the options provided by the different 
user interface buttons are as follows. Forward button 245 causes the display to 
advance to, and play, the next audiophoto. Backward button 246 causes the display to 
move to, and play, the previous audiophoto. Stop button 247 causes the soimd 

20 passage to stop playing. Linking/unlinking button 243 is for linking or imlinking the 
displayed elements: for an audiophoto, the display at any one time will be showing an 
image element 231 and a sound element 232 that are linked to each other. Pressing 
the linking/unlinking button 243 with an audiophoto selected will generally be used to . 
enable the linkage between the image element 231 and the sovmd element 232 to be 

25 broken: the result could be separated image and sound, sepm-ated audiophoto and 
sound, separated audiophoto and image, or even two separated audiophotos (if both 
the image and the sound have additional attachments) - it can also be used to modify 
the link. Both deletion and modification of links are discussed below with reference 
to Figure 3b. The deletion button 242 in this mode causes deletion of the selected 

30 audiophoto (confirmation by pressiag a different designated key would be required in 
an effective practical interface), and marker button 244 is used to mark the soxmd 
passage 232, as is discussed fiirther below with reference to Figure 3b. 



If instead the sound passage is selected, as in Figure 3b, then the meaning of the 
buttons will be different in some cases. The forward button 245 and the backward 
button 246 now cause display, and play, of the next and previous sound passage 
respectively. No change is made to the displayed image. Consequently, selection of 

5 sound passages only can be used to allow cycling through sound clips to find a clip 
for attachment to the presently displayed image. Stop button 247 is used to stop 
playback of a sound clip. Deletion button 242 is used to delete the selected sound 
passage. Linking/unlinking button 243 is now used essentially for creating or 
removing links between the selected soxmd and the currently displayed image. 

10 Pressing the button 243 causes a link to be established - or if a link akeady exists 
between the sound passage and the image, for that link to be removed. This Imk may 
represent a relationship in time between the sound passage and the image: in 
particular, for which parts of the sound passage the image is to appear. There may be 
a simple one to one relationship with no element of time (in which case, the image 

15 will shnply be associated with the beginning of the sound passage), but there may be a 
sequence of images appearing successively at different times during the sound 
passage. To support this, the user interface provides means on creation of a link to 
position that link within the sound passage. When the link is created, it will appear 
provisionally at the start of the sound passage (unless capture was contemporaneous - 

20 see below), and can be moved forward or backward along the sound passage by use of 
markers, as described below. In a variant of this approach, if capture of images and 
sound are time stamped (either absolutely, or if contemporaneously captured images 
and sound are time stamped relative to each other), the initial position of a link 
between contemporaneously recorded soimd and images can be at the actual point of 

25 capture of the image within the sound record. 

Use of marking within the soxmd record can be used to improve construction of the 
relationship between iniages and soimd. While a sound passage is playing, pressing 
of the marker button 244 places a marker (not shown in Figure 3a or Figure 3b, but 
30 visible as a visual symbol in sound record 232) in ttie sound record at a time 
corresponding to the point in the sound record at which the marker was pressed. 
When the sound record is not playing, pressing of the marker button 244 places the 
sound cUp into a state in which the markers can be used for editing. Initially, a 
passage in the sound record up to the point of the first marker is selected. Pressing 



forward button 245 advances selection to the &st marker itself, and furttier pressing 
forward button 245 advances selection to the passage of sound between the first and 
second markers. Pressing the button again advances to the second marker, and 
successive presses alternate between selection of markers and intervening passages 

5 until the end of the soimd clip is reached, at which point the process starts again fi-om 
the beginning of the clip. In tins way, either a marker itself, or the sound passage 
between two markers, can be selected. By use of the backward button 246 it is 
possible to cycle through the markers and intervening passages in the opposite 
direction. In this way, a marker or intervening passage may be selected and either 

10 deleted with delete button 242 (and so edit the sound clip itself) or used to set up a 
link with an unage (by using linking/unlinking button 243), the link thus establishing 
either when the image will display within the resulting audiophoto (if the link is with 
a marker) or over what period of time within the resulting audiophoto that the image 
will display (if the link is with an interval). 

15 

The alternative to selecting the audiophoto or selecting the sound passage is to select 
the image, and this is shown in Figure 3c. The forward button 245 and the backward 
button 246 now cause display of the next and previous images respectively. No 
change is made to the displayed sound passage. Consequently, selection of images 
20 only can be used to allow cycling through all the images for attachment to the 
presently displayed sound passage. Deletion button 242 is used to delete the selected 
image. Linking/unlinking button 243 is now used essentially for linking the selected 
image with the currently displayed soimd, using principles discussed above in relation 
to Figure 3b. 

25 

Although not shown here, practical embodiments will require a confirmation step for 
significant events, such as deletion of an audiophoto, image or sound cUp. 
Advantageously, an "undelete" fimction will also be provided to allow deletions to be 
recovered to whatever extent is practical. 

30 

A highly advantageous feature of the embodiment of the present invention is that it 
supports audiophotos that are more than a simple association of a sound passage with 
an image: in particular, an audiophoto may comprise a sound passage with one or 
more images, each of which has a relationship in time with the sound passage, or 



alternatively a collection of more than one sound passage with one or more images. 
An example of an audiophoto with a single sound passage but a plurality of images is 
shown in Figures 4 and 5. Figure 4 shows the initial view of such an audiophoto, with 
the audiophoto selected but the soxmd not playing. There are three images 42, 43, 44 
5 associated with the soimd passage represented by trace 41, each of which are 
represented in the trace 41 by a discrete clip icon 45, 46, 47. The clip icons are 
located in positions corresponding to the point in the soxmd passage at which they are 
intended to appear. The change in the display as the sound passage plays is shown in 
Figures 5a to 5c. 

10 

cup icon 45 is located at the start of the soxmd passage, so image 42 is displayed from 
the beginning of the soxmd passage. A visxxal representation for the amoimt of the 
soxmd passage so far played is provided by a backgroxmd bar 48 behind trace 41. In 
Figure 5a, it can be seen that the soxmd passage is part played, but that the second cUp 

15 icon 46 has not yet been reached. In consequence, the first image 42 is being 
displayed. When the second cUp icon 46 position is reached, then the second image 
43 will be displayed. Figxire 5b shows display of the second image 43 at a point 
during playback of the sound passage after the second clip icon 46 has been reached 
but before the third clip icon 47 has been reached. After the third clip icon 47 has 

20 been reached, the third image 44 is displayed. In this case, each of the images is 
associated with a point in time (indicating the beginning of display of that image) 
rather than a period of time (as woxild be the case if the image was linked, throu^ use 
of markers, with a part of a sound clip). There is no basic difference between the two 
situations - however, linking with passages of time can be used to provide dark 

25 intervals between pictures, or overlaps. 

It is desirable for there to be a view of the various available audiophotos and elements 
which renders it easier to compare different elements, or to move between elements to 
be edited. Accordingly, the editing mode may also provide an index view for viewing 
30 multiple elements, or collections of elements. Such an index view 61 is illustrated in 
Figure 6. This index view includes all elements that are not part of an existing 
audiophoto, together with existing audiophotos. A selection frame 62 is provided to 
allow selection for viewing and playing (as in Figure 3a to 3c) of a particular element 
or audiophoto - such selection can be achieved through use of forward button 245 and 



li 

backward button 246. Delete button 242 is used to delete the elements within 
selection frame 62. An appropriate button combination is used to switch between 
single audiophoto view and index view. When in index view, transition to single 
audiophoto view is achieved by pressing data type selection button 241 while stop 
5 button 247 is being pressed. The reverse transition from single audiophoto view to 
index view can be achieved by the same button combination. When switching from 
index view to single audiophoto view, the display 23 will change to contain the 
element or elements within the selection frame 62 in single audiophoto view. 

10 It is possible from the index view to determine whether a particular entry is an image, 
a sound passage, a simple audiophoto or a more complex audiophoto (not a simple 1:1 
relation of image and sound). Images 64 and sound passages 65 are readily 
identifiable as they contain no soxmd trace and no image respectively. Simple 
audiophotos 63 can be seen to contain one sound trace and one image. More complex 

15 arrangements, such as a single image with two sound clips 66 or a single sound cUp 
with multiple images 67, are agaia clearly identifiable as instead of providing a 
representation indicative of the start (or any other point) of the sound passage, an icon 
is provided for every element. For example the image 66 with two sound clips has a 
single image shown but two sound traces shown: however, the image 67 with a single 

20 sound clip but three images shows all three images in the image view 67, rather than 
just a single image associated with a particular point in time. The image view is thus 
an effective way to determine quickly the composition (at least the image 
composition) of existing audiophotos. 

25 Different forms of image view are possible. An altemative could be to show all 
images, or all sound samples, or all audiophotos, and to toggle- between these 
altematives with the data type selection button 241. Another altemative would be to 
provide separate index displays of sound passages and images. Images and sound 
samples could be displayed according to their time of capture (particularly appropriate 

30 if some form of absolute time stamping is available). 



By pressing button 26, the camera is switched between editing mode as discussed 
above, and capture mode. The initial image in capture mode will be as for a 
conventional digital camera: a simple image indicating what would be recorded if at 



that point image capture button 251 were to be pressed. If the image capture button 
251 is pressed at this point, then a simple image results (Uke images 64 in Figure 6). 
However, the position changes when recording of a sound passage is initiated by 
pressing sound capture button 252. This causes the display to appear as shown in 
5 Figure 7a, with a sound bar 71 showing the build up of the soimd passage (the length 
of sound so far captured, together with an indication that recording is in progress, are 
also provided) and a viewfinder image 72 below. If soimd recording stop button 253 
is pressed before image capture button 251 is pressed, then a pure soimd passage will 
be recorded, like sound passage 65 in Figure 6. However, if image capture button 251 
10 is pressed while sound recording is building up, then a contemporaneous soimd and 
image capture occurs. Advantageously, this Unks the captured image and the sound 
passage 71 directly, with the link registered for initial purposes at the point dimng the 
sound passage at which the image capture occurred (of course, the time relationship in 
the link can be changed at a later point through editing). 

15 

If an audiophoto is built up in this way, it is desirable for an alternative view to be 
available to illustrate all the elements of the audiophoto. Accordingly, if ttie data type 
selection button 241 is pressed during capture, then the image display is toggled 
between the viewfinder image 72 shown in Figure 7a to a composite of captured 
20 image thmnbnails 73, each marked with their time of recording relative to the start of 
the sound passage. The representation of the sound passage 71 is marked in this view 
with iudicators 74 showing the time of recording for the different images. 

In addition to pure capture and pure review, it is desirable and possible to support an 
25 intermediate arrangement in which new audio or image data can be inserted 
selectively into the existing set, rather than simply added to the end. 

Insertion of such an additional image or sound segment is here carried out firom the 
review mode, with the appropriate context (audiophoto, sound clip, or image) for the 
30 insertion being selected in the single audiophoto view on display 23. 

When the insertion context is a sound clip, the user presses image capture button 251 
to attach a new photo to the cUp, or presses sound capture button 252 twice to capture 
a new sound passage which is appended to the end of the existing sound clip. If the 
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user wishes to insert a new sound segment at an appropriate point during the existing 
sound passage, this can be done by advancing to and marking with marker button 244 
the desired insertion point, and in that state capturing the new sound chp. 

5 When the insertion context is a still image without sound, the user can simply press 
the image capture button 251 to add a new image after the existing image, or use the 
sound capture button 252 to record an audio segment which is then attached to the 
selected image. 

10 When the insertion context is an existing audiophoto, the user can press the image 
capture button 251 to attach a new image to the sound cUp, creating or extending a 
series of photos for the same clip. The point in the sound passage at which the new 
image is introduced can be automatically assigned, or manually positioned by use of 
the marking button 244 as previously described. Altematively, the sound capture 

15 button 252 can be used to record a further sound segment, either to append to the 
audioclip, or to position within the audioclip by use of the marking button 244 as 
previously described. 

After capture, the following information is stored in the memory 107: images (stored 
20 as discrete files), sound passages (stored as discrete files), and timestamping and 
(where relevant) linking information. There are two possible approaches for this, 
either of which can be used in satisfactory embodiments. The first is for the image 
and sound to be stored in their own separate data types, with all linking, timing and 
selection information stored in a separate index file. This solution is probably the 
25 simplest to implement, but is not convenient for selective transfer of information from 
memory 107 to another device. The second approach is for there to be a multimedia 
filetype for audiophotos, supporting image, sound, timing and linking information. If 
such a filetype is used, then this filetype can be used for all information captured 
(including images alone, and somds alone) so all files processed by the camera can be 
30 of the same filetype. This solution allows each resultant audiophoto to be fireestanding 
as a single file, without reference to an index, rendering selective transfers firom 
memory 107 straightforward. This information can then be transmitted in any of the 
paths shown in Figure 1: to a personal computer 2 (an effective route for complex 
audiophotos - which coidd then be transmitted, for example, by electronic mail), 
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directly to a printer 3 or to a reader appliance 4. For simple images, soxrnd passages, 
or audiophotos, sending directly to a printer or reader appliance is essentially as 
described in copending European Patent Application No. 98305436.2, but differences 
are required if there are multiple images in an audiophoto. For example, a complex 

5 audiophoto sent directly to a printer could be rendered with the visual element shown 
in a composite form as shown in Figure 6. The reader appliance 4 could also be 
replaced with a composite reader/viewer appliance (not shown) capable of displaying 
images and playing back sound to play back an audiophoto as captured or devised at 
the camera 1 : the functions to be provided need only be those offered by the user 

10 interface of the camera 1 (possibly retaining the browsing features but not the editing 
features), though with preferably sufficiently high quality display and sound playback 
to provide a satisfactory experience to the user. 

Li addition to still images and sound, it is possible for other data types to be captured 
15 at a digital camera. A digital camera can typically be adapted to capture and store 
short bursts of video without difficulty: such video bursts may be provided in the 
same manner as sound passages as an augmentation of a still image. Similarly, means 
to provide a graphic input at the user interface (for example, a pen-based digitiser 
interface and means for recognising handwriting or a custom alphabet such as 
20 Graffiti) may be used to provide graphic input to provide a similar augmentation. 
Such augmentations may be treated in essentially the same way as sound passages, 
and may be used as further (or conceivably alternative) elemmts in a linked group. 

As the skilled man will appreciate, many variations upon the features indicated above 
25 can be made in devising altemative embodiments of the invention. For example, the 
structiu-al arrangement of the camera can be changed considerably - display 23 may be 
used only as a user interface display, with a separate viewfinder being placed 
elsewhere on the camera, or a different arrangement and type of controls can be 
employed for the user interface. 



CLAIMS 
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1 . A digital camera comprising: 

camera apparatus for capturing still images; 

soxmd recording apparatus for capturing sound passages; 

a memory for retaining still images as still image files and sound 

passages as sound passage files; and 

a user int^face adapted for selectively linking one or more still image 
files with one or more sound passage files to form a linked group, and 
for selectively unlinking one or more files fi*om a linked group. 

2. A digital camera as claimed in claim 1, wherein a linked group may consist of 
one still image file with one or more sound passage files. 

3. A digital camera as claimed in claim 1, wherein a linked group may consist of 
one sound passage file with one or more still image files. 

4. A digital camera as claimed in any of claims 1 to 3, wherein the user interface 
is adapted such that a user can select a first and second view of the image files, 
wherein in the first view a plurality of sound passage files, still image files or 
linked groups are displayed in an index format for review, and wherein in the 
second view an individual sound passage file, still image file or linked group 
is displayed for editing. 

5. A digital camera as claimed in claim 4, wherein in the second view an 
additional sound passage file, still image file or linked group is also 
displayable, wherein the user interface is adapted to allow linkage between the 
displayed files to form a new linked group. 

6. A digital camera as claimed in claim 4 or claim 5, wherein in the second view 
a further sound passage or still image may be captured and linked 



automatically to the individual sound passage file, still image file, or linked 
group displayed. 

7. A digital camera as claimed in any preceding claim, wherein the digital 
5 camera is adapted such that a still image file may be associated with a 

particular point of time or period of time in the sound passage of a sound 
passage file. 

8. A digital camera as claimed in any preceding claim, wherein the digital 
10 camera is adapted such that the camera apparatus may capture one or more 

still images at the same time that the sound recording apparatus is capturing a 
sound passage. 

9. A digital camera as claimed in claim 8, wherein any still images captured at 
15 the same time as a sound passage are provided initially as part of a linked 

group with the sound passage. 

10. A digital camera as claimed in any preceding claim, further comprising a time 
recording means to mark a captured still image or a captured soimd passage 

20 with the time of recording. 

11. A digital camera as claimed in claim 10 where dependent on claim 8, wherein 
the time recording means is adapted to mark the time of capture of a captured 
still image relative to a contemporaneously recorded sound passage. 

25 

12. A digital camera as claimed in any preceding claim, wherein the user interface 
is adapted for selective deletion by a user of a still image file, a soxmd passage 
file or part of a sound passage file, or a linked group. 

30 13. A digital camera as claimed in any preceding claim, wherein the user interface 
is adapted such that sound passage files and still image files of a linked group 
are viewable, selectively, either as part of the linked group or uidependently of 
the linked group. 
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14. A digital camera as claimed in claim 13, wherein the user interface is adapted 
so that the sound passage files, the still image files, and the linked objects are 
all presentable to the user as a separate sequence, and that the user may select 
between these separate sequences for browsing. 

5 

15. A digital camera as claimed in any preceding claim, wherein the memory 
retains linked groups as separate sound passage files, still image files, and one 
or more index files containing linking information. 

10 16. A digital camera as claimed in any preceding claim, wherein the memory 
retains linked groups as multimedia files, containing sound passage 
information, still image information, and hnking information. 

17. A digital camera as claimed in claim 16, wherein the sound passage files and 
15 the still image files are both multimedia files, 

18. A digital camera as claimed in any preceding claim, further comprising a 
further data recording device for recording of another data type, and wherein 
the memory device is adapted to record the other data type as a further data 

20 type file, and wherein the user interface is adapted for selectively linking one 

or more further data type files into a linked group, and for selectively 
unlinking one or more further data type files from a linked group. 

19. A digital camera as claimed in claim 18, wherein the further data type is 
25 handwriting data, and the further data recording device is a device for 

recording handwriting in a digital representation. 

20. A digital camera as claimed in claim 18, wherein the further data type is a 
video clip, and the further data recording device is a device for recording 

30 video cUps. 



21. 



A digital camera as claimed in any preceding claim, wherein the sound 
recording apparatus and the camera apparatus are arranged for data capture in 
a substantially common direction. 
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22. A digital camera as claimed in any preceding claim, wherein the memory is a 
flash memory device. 

5 23. A digital camera as claimed in any preceding claim, wherein the digital 
camera is adapted to be handheld by a user for still image and sound passage 
capture. 

24. A program for operating the user interface of a digital camera comprising 
0 camera apparatus for capturing still images and sound recording apparatus for 

capturing sound passages, wherein the program comprises means for linking 
one or more still images with one or more sound passages to form a hnked 
group, and means for unlinking one or more elements from a Unked group. 
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ABSTRACT 

30990001 

DIGITAL CAMERA WITH SOUND RECORDING 

5 

A digital camera 1 for capturing sound passages as well as still images has the 
following elements. It has a camera apparatus 21 for capturing still images. It also has 
a soxmd recording app^atus 101, 102, 103, 106 for capturing sound passages. A 
memory 107 is used for retaining still images as still image files and sound passages 

10 as soimd passage files. The user interface of flie camera is adapted for selectively 
linking one or more still image files with one or more sound passage files to form a 
linked group, and for selectively unlinking one or more files firom a linked group. 
This equivalent treatment of still images and sound passages allows effective use of 
the camera for recording ambient sound together with images, and for flexible 

15 constmction of sound/image composites representing a user's experience in a rich and 
effective manner. 
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